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Abstract 

This  effort  focused  on  developing  EEG-derived  indicators  of  verbal  versus  spatial  working  memory  load.  A 
wireless  EEG  headset  acquired  data  during  execution  of  both  simple  and  complex  tasks  associated  with  a  Tactical 
Tomahawk  Weapons  Control  System  (TTWCS).  The  results  established  the  feasibility  of  characterizing  EEG 
correlates  specific  to  verbal  and  spatial  working  memory.  The  next  goal  is  to  leverage  these  real-time  working 
memory  indices  as  a  feedback  loop  to  direct  closed-loop  human-system  interaction.  Specifically,  if  the  preliminary 
EEG  indices  derived  in  this  study,  in  combination  with  other  physiological  or  behavioral  inputs,  are  shown  to  relate 
to  the  degree  of  working  memory  overload  in  the  TTWCS  or  similar  tasks,  they  could  provide  a  valuable 
contribution  to  real-time  adaptive  aiding  of  human-system  interaction. 


1  Introduction 

Working  memory  overload  is  one  of  the  key  contributors  to  operator  errors  during  complex  tasks.  The  capacity  of 
human  working  memory  has  been  defined  as  the  number  of  items  that  can  be  held  in  conscious  attention  for  use  in  a 
specific  task  or  for  later  long-term  storage  (Baddeley  &  Logie,  1999).  The  constraints  of  working  memory  are 
particularly  relevant  during  skill  acquisition  where  working  memory  capacity  is  frequently  exceeded.  Traditional 
models  (c.f.  Baddeley  &.  Hitch,  1974)  characterize  working  memory  as  having  two  separate  and  relatively 
autonomous  subsystems;  verbal  (i.e.,  phonological  loop)  and  spatial  (i.e.,  visuo-spatial  sketchpad),  but 
contemporary  models  suggest  further  disassociation  (potentially  based  on  sensory  modality)  (c.f.  Miyake  &  Shah, 
1999).  Recent  investigations  suggest  that  working  memory  capacity  can  be  enhanced  by  utilizing  verbal,  spatial,  or 
alternative  sensory  modalities  in  a  complementary  manner  (Wickens,  2002). 

The  development  of  an  effective  method  for  monitoring  working  memory  load  and  delineating  the  verbal  and  spatial 
components  could  greatly  enhance  the  speed  and  efficiency  of  human-system  interaction.  A  real-time  monitor  could 
identify  periods  of  spatial  and/or  verbal  working  memory  overload  and  provide  adaptive  aiding,  such  as  switching 
from  verbal  or  spatial  presentation  formats,  to  meet  operator  requirements  (Schmorrow,  Stanney,  Wilson,  &  Young, 
2005).  However,  frequent  switching  between  one  modality  or  task  and  another  may  incur  a  “cost”  due  to  stimulus 
competition,  ambiguity,  or  distractions  (Baddeley,  2003;  Matlin,  1998).  Thus,  it  is  essential  to  develop  an 
understanding  of  how  best  to  leverage  the  multimodal  capacity  of  working  memory  without  incurring  such  costs. 

As  a  first  step  towards  developing  a  real-time  neurophysiological  working  memory  index  that  could  trigger  such 
adaptive  aiding,  a  wireless  electroencephalographic  (EEG)  system  was  used  to  acquire  data  during  working  memory 
tasks  of  varying  complexity.  In  the  first  phase  of  the  study,  participants  performed  three  simple  working  memory 
tasks;  one  spatial  and  two  verbal,  designed  in  accordance  with  methods  previously  reported  (Proffitt,  2003).  The 
simple  verbal  and  spatial  tasks  used  elements  of  a  Tactical  Tomahawk  Weapons  Control  System  (TTWCS)  so  that 
they  could  be  integrated  into  a  more  operationally  relevant  simulated  TTWCS  environment  that  would  serve  as  the 
testbed  for  the  second  portion  of  the  study.  The  rationale  for  this  design  was  that  if  a  set  of  EEG  parameters  were 
identified  that  distinguished  between  the  simple  verbal  and  spatial  tasks,  these  parameters  could  then  be  evaluated  in 
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the  more  complex  TTWCS  testbed.  The  initial  goal  of  the  first  phase  was  to  select  a  set  of  EEG  variables  optimized 
for  discrimination  of  verbal  from  spatial  processing  in  the  simple  tasks  (Proffitt,  2003). 

The  second  phase  of  the  study  was  designed  to  evaluate  the  utility  of  the  EEG  variables  in  a  complex  TTWCS 
simulation  task  and  to  determine  whether  additional  EEG  parameters  were  required  to  provide  accurate  delineation 
of  the  spatial  and  verbal  processing  within  the  more  complex  task  environment.  The  selected  EEG  variables  would 
then  serve  as  inputs  to  a  neural  net  that  used  a  combination  of  physiological  parameters  to  identify  relevant  cognitive 
state  changes  while  operators  performed  the  TTWCS  simulation  tasks.  The  outputs  of  the  neural  net  controlled  the 
timing  and  introduction  of  multimodal  augmentation  strategies  designed  to  optimize  the  distribution  of  perception 
and  cortical  processing  within  the  TTWCS  environment  to  take  advantage  of  the  totality  of  human  capacity  for 
multimodal  communication.  These  studies  represent  the  initial  steps  in  building  the  foundation  for  designing  a 
robust  Command  and  Control  (C^)  system  that  can  adapt  interaction  techniques  to  meet  specific  user  conditions. 

2  Methods 

The  current  study  was  performed  in  two  phases.  During  the  first  phase,  EEG  data  was  acquired  while  participants 
performed  simple  working  memory  tasks.  In  the  second  phase,  participants  performed  a  complex  working  memory 
task,  playing  the  role  of  a  Tactical  Strike  Coordinator  interacting  with  a  simulated  TTWCS  environment. 

2.1  Participants 

For  the  Simple  Working  Memory  Task  experiment,  12  Lockheed  Martin  employees  were  studied.  The  participants 
were  provided  verbal  instructions  but  no  hands-on  training  prior  to  the  start  of  each  task.  The  tasks  were  presented 
in  the  following  order;  Missile  ID,  Mental  Addition  and  Missile  Location.  Data  from  two  participants  were 
excluded  from  analyses.  One  participant  was  dropped  due  excessive  EMG  (muscle)  artifact  in  the  mono-polar 
channels  due  to  the  mastoid  reference  electrode.  The  second  participant  was  dropped  due  to  an  inability  to 
determine  the  start  and  end  of  each  task  due  to  missing  time  synchronization  data. 

Of  the  ten  Lockheed  Martin  employees  who  participated  in  the  Complex  Working  Memory  Task  study,  data  from 
seven  participants  were  included  for  analyses.  Two  participants  were  dropped  due  to  excessive  EMG  artifact  in  the 
monopolar  channels  due  to  the  mastoid  reference  electrode.  The  third  participant  was  dropped  due  to  an  inability  to 
determine  the  start  and  end  of  each  task  due  to  missing  synchronization  data.  Participants  were  provided  a  fixed 
timeframe  for  training  on  the  TTWCS  tasks  in  advance  of  their  testing  session. 

2.2  Simple  Working  Memory  Tasks 

The  three  simple  working  memory  tasks  were  designed  to  replicate  experiments  performed  by  Proffitt  (2003).  The 
tasks  were  designed  to  tap  into  both  verbal  and  spatial  working  memory,  such  that  they  would  provide  differentiable 
EEG  signatures. 

2.2. 1  Verbal  Task  -  “Missile  ID  ’’ 

In  this  auditory  recognition  task,  participants  were  presented  with  synthesized  speech  listing  a  set  of  two  Missile 
Identifiers  (ID)  (e.g.  56U,  ISP).  After  the  set  of  Missile  ID’s  were  presented,  the  computer  cued  the  user  by  saying 
the  word  “listen”  and  then  “spoke”  a  series  of  Missile  IDs.  Participants  responded  via  a  keyboard  as  to  whether  or 
not  a  spoken  Missile  ID  matched  one  of  the  sets  previously  presented. 

2.2.2  Verbal  Task  -  “Mental  Addition” 

In  this  computation  task,  participants  were  presented  with  a  display  containing  a  single  number  at  the  center  of  the 
display  and  instructed  to  respond  to  the  number.  A  new  number  was  presented  each  time  the  participant  responded. 
Participants  were  instmcted  to  add  a  series  of  numbers  until  prompted  to  report  the  total. 


2.2.3  Spatial  Task  -  “Missile  Location” 


In  this  “grid-task”,  participants  were  presented  with  a  5  x  5  grid  that  contained  from  three  to  five  missiles.  The 
display  was  shown  for  a  brief  interval  and  then  removed.  After  approximately  40  seconds,  the  grid  reappeared  with 
a  subset  of  the  missiles  shown.  Participants  were  instructed  to  indicate  the  locations  of  missing  missiles. 
Participants  had  approximately  4  seconds  to  indicate  locations  of  missing  missiles  before  the  grid  disappeared. 

2.3  Complex  Working  Memory  Tasks 

In  the  Complex  tasks  the  operator  performed  the  role  of  a  Tactical  Strike  Coordinator  interacting  with  a  simulated 
TTWCS  environment.  It  is  the  job  of  the  Tactical  Strike  Coordinator  to  assign  missile  strikes  to  specific  targets,  and 
to  monitor  and  reassign  missiles  to  “emergent  targets”  as  these  events  occur.  The  scenarios  used  to  select  EEG 
correlates  included  performing  assessment  of  missile  coverage  zones,  referred  to  as  the  “Location  Task”,  and 
retargeting  missiles  based  on  emerging  targets  of  higher  priority,  referred  to  as  the  “Retarget  Task”. 

2.3.1  Spatial  Task  -  “Location” 

The  location  task  (Eigure  1)  was  separated  into  three  parts;  encoding,  rehearsal,  and  recall.  During  the  “encoding” 
period,  participants  were  given  15-seconds  to  study  the  location  and  10-minute  coverage  zone  (i.e.,  circular  region 
around  each  missile  shown  in  Eigure  1  below)  of  each  missile  and  the  associated  targets.  During  the  45-second 
“rehearsal”  period,  the  coverage  zone  circles  were  removed,  however  participants  were  provided  the  opportunity  to 
continue  memorizing  the  initial  information  that  was  previously  displayed.  During  the  “recall”  period,  participants 
were  provided  30-seconds  to  identify  targets  that  were/were  not  covered  by  any  missile’s  coverage  zones.  Since 
most  participants  completed  their  “recall”  responses  well  within  the  allotted  time,  only  the  first  12  seconds  of  the 
recall  period  were  analyzed. 

sn  icorej 


Figure  1:  Screen  presentation  during  location  task 

2.3.2  Verbal  Task  -  “Retarget” 

The  “Retarget”  task  (Eigure  2)  required  participants  to  retarget  missiles  based  on  higher  priority  emergent  targets. 
Participants  were  provided  10  minutes  to  reallocate  missile  coverage  to  as  many  emergent  targets  as  possible,  while 
maintaining  coverage  on  as  many  high  and  medium  default  targets  as  possible.  There  were  four  rules  for  retargeting 
missiles;  a)  missile  warhead  types  must  match  target  warhead  types,  b)  the  number  of  missiles  servicing  a  target 
must  match  the  number  of  missiles  required  by  that  target,  c)  highest  priority  emergent  targets  should  be  retargeted 
first,  d)  only  Loiter  or  Retarget  (L/R)  missiles  may  be  retargeted.  The  information  available  for  retargeting  was 
presented  in  the  text  below  each  missile  and  target.  Emergent  targets  were  colored  red  and  their  appearance  and 
locations  were  randomly  assigned  throughout  the  task.  The  Task  Readout,  to  the  right  of  the  missile-target  map. 
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provided  information  to  the  participant  helpful  in  determining  a  retargeting  strategy.  This  information  included  the 
amount  of  time  from  missile-target  intercept  and  closest  missile  to  emergent  target. 
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Figure  2:  Screen  presentation  during  retarget  task 


2.4  Data  Acquisition 

2.4.1  Tactical  Tomahawk  Weapons  Simulation  (TTWCS)  Test  bed 

The  TTWCS  simulation  test  hed  ran  on  an  Intel  Pentium  4-  3.0  GHz  (800MHz  FSB  -  1MB  Cache)  computer  with  1 
GB  (2  64-hit  wide  DDR  data  channel)  memory,  a  60G  7200rpm  high  performance  drive,  and  an  ATI  Radeon 
Mobility  Integrated  256MB  DDR  Video,  with  Accelerated  OpenGL  and  8x  ultraAGP.  The  operating  system  was 
Red  Hat/Fedora  Linux  with  customized  2.6.9  kernel  (Fedora  Core  2  with  latest  updates).  The  visual  interface  was 
presented  on  a  17”  WSXGAh-  (1680x1050)  screen.  All  user  responses  were  with  standard  keyboard  and  2-button 
mouse.  The  audio  was  presented  via  6  channel  desktop  speakers  based  on  AC’97  2.2  virtual  6-channel  output 
onboard  sound. 

2.4.2  EEG  Acquisition 

A  modified  wireless  EEG  headset  acquired  data  from  9  monopolar  sites  referenced  to  linked  mastoids  (E3,  E4,  Ez, 
C3,  C4,  Cz,  P3,  P4,  POz)  and  2  bipolar  sites  (Cz-POz,  Ez-POz)  (see  Berka  2004  for  basic  EEG  headset  details). 
The  EEG  transceiver  unit  was  interfaced  to  a  laptop  computer  (Pentium  2.4  MHz  with  512  RAM  and  a  Windows  XP 
operating  system)  that  operated  the  EEG  acquisition  software  used  to  generate  an  EEG  record  and  provide  time 
synchronization  data  to  the  TTWCS  test  bed  via  a  TCP-IP  network  protocol. 

2.5  Data  Reduction 

Eor  the  Simple  Working  Memory  Tasks,  approximately  30  one-second  epochs  of  EEG  data  per  task  were  extracted 
from  the  three  task  conditions  (i.e..  Missile  ID,  Mental  Addition,  and  Missile  Location).  Eor  the  Complex  Working 
Memory  Tasks,  seventy  one-second  epochs  were  extracted  for  the  training  data  set  from  each  of  two  segments;  the 
encoding  segment  of  the  “location”  task  and  the  retargeting  segment  of  the  “retarget”  task.  The  balance  of  data  from 
the  tasks  was  used  for  testing  discriminant  function  models  (see  section  2.5). 

Processing  of  the  EEG  included  identification  and  decontamination  or  rejection  of  artifacts,  including  eye  blinks, 
spikes,  saturation,  excursions,  and  EMG,  using  previously  published  procedures  (Berka  2004).  The  EEG  power 


spectra  for  each  1  Hz  bin  between  3  -  40  Hz  were  computed  for  each  channel,  generating  a  total  of  418  EEG 
variables  from  1 1  channels  for  each  one-second  epoch. 

One  of  the  long-term  goals  of  the  project  is  to  optimize  the  selection  of  sensor  locations,  number  of  channels  and 
number  of  EEG  variables  to  provide  accurate  classifications  while  minimizing  the  amount  of  data  acquired.  In 
order  to  assess  a  variety  of  headset  configurations,  a  total  of  eight  EEG  data  sets  were  created.  The  “Mid-line 
Bipolar”  data  set  included  only  the  1  Hz  bins  from  EzPOz  and  CzPOz  for  the  Simple  and  Complex  Working 
Memory  Tasks,  respectively.  The  “Mid-line  All”  data  sets  included  the  bi-polar  sites  plus  the  mono-polar  sites  Ez, 
Cz  and  POz.  The  “Lateral”  data  sets  included  the  1  Hz  bins  from  the  mono-polar  sites  E3,  E4,  C3,  C4,  P3  and  P4. 
The  “Mid  -  Eat”  data  sets  included  all  bi-polar  and  mono-polar  data  from  all  1 1  channels. 

2.6  Variable  Selection  and  Model  Development 

Three-  and  two-class  discriminant  function  models  were  used  to  classify  data  from  the  Simple  and  Complex 
Working  Memory  Tasks,  respectively.  Step-wise  analysis  was  used  to  select  predictive  variables  for  each  of  the 
eight  data  sets.  The  discriminant  function  models  generated  with  variables  and  coefficients  derived  from  the  Simple 
(Complex)  Working  Memory  Task  for  each  data  set  were  applied  to  the  Complex  (Simple)  Task  data  to  evaluate  the 
influence  of  variable  selection.  Individualized  discriminant  function  models  were  derived  to  assess  the  benefit  of 
fitting  coefficients  to  the  unique  EEG  patterns  of  each  participant. 

3  Results 

3.1  Classification  Models  During  the  Simple  Working  Memory  Task 

The  results  from  four  discrimant  function  models  applied  to  the  Simple  Working  Memory  Tasks  are  presented  in 
Table  1.  The  Mid-line  Bipolar  model  provided  surprisingly  good  classification  for  the  Missile  ID  and  Missile 
Location  tasks  (using  just  two  channels  of  data  and  eleven  variables),  but,  due  to  poor  classification  during  the 
Mental  Addition  task,  reported  an  overall  classification  accuracy  of  only  50%. 

The  Lateral  Model  provided  better  classification  in  the  Missile  ID  task  compared  to  the  Mid-line  All  with  a  similar 
number  of  channels  and  variables,  however  the  overall  classification  accuracies  were  similar.  The  Lateral  Model 
required  an  additional  three  sensor  sites  to  achieve  the  better  performance  in  the  Missile  ID  task.  The  Mid-Lat 
Model  was  clearly  superior,  although  nine  sensors,  1 1  channels  and  43  variables  were  required  to  achieve  these 
results.  Table  2  presents  the  number  of  variables  selected  from  each  of  the  sensor  sites  for  the  Mid  -  Eat  Model. 
The  rank  order  of  the  top  seven  variables  (partial  values  >  0.03)  was  C4  37Hz,  P4  llHz,  P3  13Hz,  CzPOz  31Hz, 
Cz  32Hz,  CzPOz  18Hz,  and  C4  26Hz. 


Table  1:  Classification  Accuracy  of  Three-Condition  Simple  Task  Model 


Model 

# 

Sensors 

# 

Channels 

# 

Variables 

Percent  of  epochs  correctly  classified 

Class 

Accuracy 

Verbal  - 
Missile  ID 

Verbal- 

Mental 

Addition 

Spatial  - 

Missile 

Location 

Mid-line 

Bipolar 

3 

2 

11 

60.4 

40.6 

61.5 

50.8 

Mid-line 

All 

3 

5 

22 

65.7 

60.7 

67.4 

64.5 

Lateral 

6 

6 

26 

75.2 

60.3 

65.1 

66.9 

Mid  -  Eat 

9 

11 

43 

77.8 

76.4 

76.6 

77.0 

Table  2:  Mid  -  Eat  Simple  Model:  Number  of  Variables  Selected  by  Sensor  Site 


Site 


Monopolar 

Left 

Right 

Midline 

Bipolar 


Erontal 

1 

5 

0 

EzPOz 

5 

Central 

6 

5 

3 

CzPOz 

6 

Parietal 

2 

6 

3* 

B-Alert  Class 

1 

Totals 

9 

16 

6 

12 

*  The  midline  sensor  is  parietal  occipital  (POz) 


3.2  Classification  Models  During  the  Complex  Working  Memory  Task 

The  classification  distribution  trends  across  the  four  models  for  the  Complex  Tasks  were  similar  to  the  Simple  Task 
with  the  Mid-Lat  sites  providing  optimal  classification  accuracy.  For  this  reason,  only  results  from  the  Mid-Lat 
Model  are  reported.  Interestingly,  the  number  of  variables  selected  to  discriminate  the  Complex  Tasks  was 
significantly  reduced  compared  to  the  Simple  task  from  43  to  11  (Table  3  and  5).  The  rank  order  of  variables  with  a 
partial  values  >  0.03  were  P4  27Hz,  P4  36Hz,  and  Fz  12Hz.  Consistent  with  the  Simple  Task,  the  largest  number 
of  variables  was  selected  from  the  lateral  right  region  (Table  3).  Table  4  presents  the  classification  accuracy  of  the 
Two-Class  Mid-Lat  model  with  variables  selected  from  the  Complex  Task  (second  row  of  data)  applied  to  the  three 
segments  of  the  Location  task  and  two  segments  of  the  Retarget  task. 


Table  3:  Mid  -  Lat  Complex  Model:  Number  of  Variables  Selected  by  Sensor  Site 


Site 

Monopolar 

Bipolar 

Left 

Right 

Midline 

Erontal 

1 

1 

0 

EzPOz 

0 

Central 

1 

2 

2 

CzPOz 

1 

Parietal 

0 

2 

1* 

B-Alert  Class 

0 

Totals 

2 

5 

3 

1 

*  The  midline  sensor  is  parietal  occipital  (POz) 


3.3  Influence  of  Variable  Selection  on  Classification  Models 

Due  to  the  substantial  difference  in  the  number  of  variables  selected  for  the  Simple  vs.  Complex  Tasks,  a  Three- 
Class  Simple  Task  Model  using  the  Complex  Task  variables  and  a  Two-Class  Complex  Task  Model  using  the 
Simple  Task  variables  was  developed.  The  result  in  Table  4  show  that  the  classification  accuracies  of  the  two 
models  applied  to  the  Complex  Task  data  were  relatively  similar.  However,  the  results  in  Table  5  show  that  the 
Complex  task  variables  are  unable  to  discriminate  the  Simple  Tasks  better  than  chance. 


Table  4:  Results  for  Complex  Task  Models  Using  Variables  from  Simple  vs.  Complex  Tasks 


Model 

Variable 

Type 

Percent  of  epochs  classified  Verbal  (V)  or  Spatial  (S) 

Spatial  Task  -  Location 

Verbal  Task  -  Retarget 

Encoding 

Rehearsal 

Recall 

Encoding 

Retarget 

S 

V 

S 

V 

S 

V 

S 

V 

S 

V 

Mid- 

Lat 

Simple 

65.8 

34.2 

62.6 

37.4 

51.6 

48.4 

48.0 

52.0 

37.6 

62.4 

Complex 

67.1 

32.9 

58.9 

41.1 

50.0 

50.0 

43.0 

57.0 

33.9 

66.1 

Table  5:  Results  for  Simple  Task  Models  Using  Variables  from  Simple  vs.  Complex  Tasks 


Model 

Variable 

Model 

# 

Variables 

Percent  of  epochs  correctly  classified 

Class 

Accuracy 

Verbal  - 
Missile  ID 

Verbal- 

Mental  Addition 

Spatial  - 

Missile  Location 

Mid  and  Lat-  All 

Simple 

43 

77.8 

76.4 

76.6 

77.0 

Mid  and  Lat-  All 

Complex 

11 

53.5 

55.0 

39.5 

49.5 

To  determine  if  the  large  difference  in  the  number  of  simple  versus  complex  variables  required  to  discriminate 
between  the  tasks  was  caused  by  the  additional  task  in  the  Simple  Model,  an  alternative  Two-class  model  was 
evaluated  to  compare  only  the  verbal  (“Missile  ID”)  and  spatial  (“Missile  Location”)  sub-tests  of  the  Simple  Tasks. 
This  model  required  34  EEG  variables  to  accurate  classify  the  two  states,  still  significantly  more  variables  than  the 
1 1  required  for  the  Complex  Task. 


3.4  Fitting  Model  Coefficients  to  Accommodate  Individual  Differences  in  EEG 


Table  6  presents  the  classification  accuracies  for  individual  participants  using  the  Two-Class  Model  with  the  Simple 
Task  variables  and  group  discriminant  function  coefficients.  These  results  demonstrate  wide  variability  in  the 
classification  accuracies.  Table  7  presents  findings  from  Two-Class  models  using  the  same  variables,  with 
discriminant  function  coefficients  fitted  to  the  individual’s  EEG  patterns. 


Table  6:  Classification  Accuracy  Two-Class  Model  Using  Group  Coefficients 


Participant 

Number 

Percent  of  epochs  classified  Verbal  (V)  or  Spatial  (S) 

Spatial  Tasks 

Verbal  Task 

Encoding 

Rehearsal 

Recall 

Encoding 

Retarget 

S-E 

V-R 

S-E 

V-R 

S-E 

V-R 

S-E 

V-R 

S-E 

V-R 

551 

48.7 

51.3 

34.7 

65.3 

22.5 

77.5 

65.9 

34.1 

54.2 

45.8 

552 

91.2 

8.8 

77.0 

23.0 

71.8 

28.2 

77.8 

22.2 

70.8 

29.2 

553 

38.9 

61.1 

55.6 

44.4 

29.6 

70.4 

13.3 

86.7 

16.0 

84.0 

554 

63.7 

36.3 

65.3 

34.7 

58.3 

41.7 

46.2 

53.8 

40.8 

49.2 

555 

73.6 

26.4 

73.4 

26.6 

57.5 

42.5 

41.7 

58.3 

24.5 

75.5 

556 

70.9 

29.1 

57.4 

42.6 

64.1 

35.9 

30.8 

69.2 

11.8 

88.2 

557 

74.6 

25.4 

74.1 

25.9 

51.4 

48.6 

53.1 

46.9 

37.1 

62.9 

Mean 

65.9 

34.1 

62.5 

37.5 

50.7 

49.3 

47.0 

53.0 

36.4 

63.6 

Table  7:  Classification  Accuracy  Two-Class  Mid-Lat  Model  Using  Individualized  Coefficients 


Participant 

Number 

Percent  of  epochs  classified  Verbal  (V)  or  Spatial  (S) 

Spatial  Task  -  Location 

Verbal  Task  -  Retar 

get 

Encoding 

Rehearsal 

Recall 

Encoding 

Retarget 

S-E 

V-R 

S-E 

V-R 

S-E 

V-R 

S-E 

V-R 

S-E 

V-R 

551 

94.1 

5.9 

89.3 

10.7 

92.5 

7.5 

27.3 

72.7 

17.7 

82.3 

552 

77.5 

22.5 

52.3 

46.7 

43.6 

56.4 

30.6 

69.4 

32.9 

67.1 

553 

93.7 

6.3 

88.9 

11.1 

88.9 

11.1 

63.3 

36.7 

21.7 

78.3 

554 

79.0 

21.0 

58.9 

41.1 

47.2 

52.8 

20.5 

79.5 

21.9 

78.1 

555 

89.3 

10.7 

79.7 

20.3 

70.0 

30.0 

47.2 

52.8 

12.4 

87.6 

556 

95.3 

4.7 

77.0 

23.0 

76.9 

23.1 

46.1 

53.9 

11.3 

88.7 

557 

88.7 

11.3 

87.0 

13.0 

32.4 

67.6 

25.0 

75.0 

15.1 

84.9 

Mean 

88.2 

11.8 

76.2 

23.7 

64.5 

35.5 

37.1 

62.9 

19.0 

81.0 

4  Conclusions 

This  paper  presents  one  model  approach  to  selecting  EEG  parameters  that  can  be  implemented  in  systems  designed 
to  monitor  and  interpret  real-time  cognitive  state  changes.  Previous  work  by  the  investigative  team  (Berka,  2004, 
Berka,  2005)  validated  this  approach  to  development  of  an  EEG-based  closed  loop  system  where  EEG  correlates  of 
mental  workload  were  first  established  and  validated  in  a  series  of  simple  tasks.  The  EEG-workload  measures  were 
subsequently  validated  in  a  complex  Aegis  simulation  environment  and  were  used  successfully  to  control  the  pacing 
of  stimulus  presentation  to  optimize  performance. 

These  data  establish  the  feasibility  of  characterizing  EEG  correlates  specific  to  verbal  and  spatial  working  memory 
in  both  simple  and  complex  task  environments.  The  variables  derived  from  this  analysis  can  be  computed  in  real¬ 
time  to  provide  a  second-by-second  assessment  of  verbal  and  spatial  processing.  The  investigators  acknowledge 
that  the  terms  “verbal”  and  “spatial”  apply  only  within  the  context  of  the  tasks  and  conditions  designed  for  the 
TTWCS  simulation  testbed.  Although  the  experimental  design  team  attempted  to  create  task  conditions  that 
required  predominantly  verbal  or  spatial  processing  to  provide  data  for  selection  of  EEG  parameters,  the  purity  of 
these  conditions,  particularly  in  the  complex  tasks  is  questionable. 


The  difference  in  the  number  of  EEG  variables  selected  to  discriminate  Simple  Task  in  comparison  to  the  number 
required  for  the  Complex  Task  (i.e.,  43  vs.  11)  suggests  that  the  two  complex  tasks  were  more  distinctive  as 
reflected  in  their  EEG  characteristics  than  the  three  simple  tasks,  so  fewer  variables  were  required.  An  alternative  2- 
class  model  was  evaluated  to  compare  only  the  verbal  (“Missile  ID”)  and  spatial  (“Missile  Location”)  sub-tests  of 
the  Simple  Tasks.  This  model  required  34  EEG  variables  to  accurate  classify  the  two  states,  still  significantly  more 
variables  than  the  1 1  required  for  the  Complex  Task.  It  is  also  possible  that  the  Simple  Task  may  have  required 
more  variables  for  accurate  classification  due  to  the  limited  attention  required  to  perform  the  sub-tasks,  resulting  in 
greater  variability  in  EEG  activity  within  and  between  individuals. 

Data  from  the  complex  tasks  reveal  that  the  patterns  of  EEG  associated  with  “location”  were  distinctive  from  those 
observed  during  “retargeting”.  It  is  possible  that  the  complex  tasks  were  more  engaging  than  the  simple  tasks  and 
elicited  overall  a  greater  percentage  of  focused  time-on-task.  Interestingly,  the  EEG  variables  selected  for  the  Simple 
Task  model  provided  similar  classification  accuracy  when  applied  to  the  Complex  Task,  but  those  selected  for  the 
Complex  Task  were  not  effective  in  discriminating  the  simple  sub-tasks.  This  finding  suggests  that  the  applicability 
of  variables  selected  for  one  task  may  or  may  not  be  universally  applied  to  other  tasks  and  should  be  investigated  as 
a  component  of  model  development. 

The  decision  on  how  to  accommodate  individual  differences  is  relevant  to  all  aspects  of  model  building  in  the  design 
of  closed-loop  systems.  Although  the  fitting  of  the  discriminant  function  coefficients  to  each  individual 
demonstrated  a  marked  improvement  on  the  consistency  of  correct  classifications  across  participants  the 
“classification  accuracies”  of  the  various  models  should  be  interpreted  with  some  caution.  Distinctive  patterns  of 
EEG  classifications  may  actually  reflect  differences  in  strategic  approach  to  the  task  demands  or  differential 
allocation  of  verbal  and  spatial  memory.  More  detailed  investigations  of  the  relationship  of  task  performance  and 
EEG  classifications  on  a  second-by-second  basis  are  required  to  better  understand  these  associations.  In  addition, 
post-test  interviews  should  be  conducted  to  determine  whether  participants  were  employing  unique  strategies  for 
completing  the  tasks. 

If  these  preliminary  data  are  replicated  and  the  EEG  indices,  in  combination  with  other  physiological  or  behavioral 
inputs,  are  shown  to  relate  to  the  degree  of  working  memory  overload  in  the  TTWCS  or  similar  tasks,  they  could 
provide  a  valuable  contribution  to  real-time  adaptive  aiding  of  human-system  interaction.  The  goal  of  the  present 
model  approach  was  to  provide  inputs  to  a  neural-net  based  system  that  would  employ  augmentation  strategies 
designed  to  take  advantage  of  the  totality  of  human  capacity  for  multimodal  communication.  More  specifically,  the 
system  involves  adaptive  multimodal  mediation  and  attention  alerting  mechanisms,  which  incorporate  multiple 
display  strategies  to  invoke  alternate  sensory  modalities  given  a  TTWCS  user’s  current  cognitive  state  as  measured 
by  real-time  biophysical  data.  Based  on  output  from  the  physiological  sensors,  as  well  as  an  understanding  of 
current  system  state  (i.e.,  which  verbal  and  spatial  tasks  are  currently  being  performed  and  their  relative  loading), 
alternate  modality  display  strategies  can  be  employed  (i.e.,  modality-based  task  switching/augmenting).  Any  such 
aiding  must  be  implemented  judiciously,  as  any  gains  realized  could  be  tempered  if  the  costs  for  modality  switching 
are  high. 
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